Method Of Signalling Motion Information For Efficient Scalable Video Compression

ABSTRACT

Embodiments of a method for incrementally coding and signaling motion information for a video compression system involving a motion adaptive transform and embedded coding of transformed video samples using a computer are disclosed herein. In one such embodiment, the method includes (a) storing computer-readable instructions in the computer which, when executed, produce an embedded motion field bit-stream, representing each, motion field in coarse to fine fashion and (b) storing computer-readable instructions in the computer which, when executed, interleave contributions from said embedded motion field bit-stream with successive contributions from said embedded coding of the transformed video samples.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.10/528,965, filed Feb. 13, 2006, which was the National Stage ofInternational Application No. PCT/AU03/01233, filed Sep. 19, 2003 andclaims priority to Australian Patent Application No. 2002951574, filedSep. 20, 2002, all of which are hereby incorporated by reference intheir entireties.

TECHNICAL FIELD

The present invention relates to efficient compression of motion videosequences and, in preferred embodiments, to a method for producing afully scalable compressed representation of the original video sequencewhile exploiting motion and other patio-temporal redundancies in thesource material. The invention relates specifically to therepresentation and signalling of motion information within a scalablecompression framework which employs motion adaptive wavelet liftingsteps. Additionally, the present invention relates to the estimation ofmotion parameters for scalable video compression and to the successiverefinement of motion information by temporal resolution, spatialresolution or precision of the parameters.

BACKGROUND

For the purpose of the present discussion, the term “internet” will beused both in its familiar sense and also in its generic sense toidentify a network connection over any electronic communications mediumor collection of cooperating communications systems.

Currently, most video content which is available over the internet mustbe pre-loaded in a process which can take many minutes over typicalmodem connections, after which the video quality and duration can stillbe quite disappointing. In some contexts video streaming is possible,where the video is decompressed and rendered in real-time as it is beingreceived; however, this is limited to compressed bit-rates which arelower than the capacity of the relevant network connections. The mostobvious way of addressing these problems would be to compress and storethe video content at a variety of different bit-rates, so thatindividual clients could choose to browse the material at the bit-rateand attendant quality most appropriate to their needs and patience.Approaches of this type, however, do not represent effective solutionsto the video browsing problem. To see this, suppose that the video iscompressed at bit-rates of R, 2R, 3R, 4R and 5R. Then storage must befound on the video server for all these separate compressed bit-streams,which is clearly wasteful. More importantly, if the quality associatedwith a low bit-rate version of the video is found to be insufficient, acomplete new version must be downloaded at a higher bit-rate; this newbit-stream must, take longer to download, which generally rules out anypossibility of video streaming.

To enable real solutions to the remote video browsing problem, scalablecompression techniques are essential. Scalable compression refers to thegeneration of a bit-stream which contains embedded subsets, each ofwhich represents an efficient compression of the original video withsuccessively higher quality. Returning to the simple example above, ascalable compressed video bit-stream might contain embedded sub-setswith the bit-rates of R, 2R, 3R, 4R and 5R, with comparable quality tonon-scalable bit-streams, having the same bit-rates. Because thesesubsets are all embedded within one another, however, the storagerequired on the video server is identical to that of the highestavailable bit-rate. More importantly, if the quality associated with alow bit-rate version of the video is found to be insufficient, only theincremental contribution required to achieve the next higher level ofquality must be retrieved from the server. In a particular application,a version at rate R might be streamed directly to the client inreal-time; if the quality is insufficient, the next rate-R incrementcould be streamed to the client and added to the previous, cachedbit-stream to recover a higher quality rendition in real time. Thisprocess could continue indefinitely without sacrificing the ability todisplay the incrementally improving video content in realtime as it isbeing received from the server.

The above application could be extended in a number of exciting ways.Firstly, if the scalable bit-stream also contains distinct subsetscorresponding to different intervals in time, then a client couldinteractively choose to refine the quality associated with specific timesegments which are of the greatest interest. Secondly, if the scalablebit-stream also contains distinct subsets corresponding to differentspatial regions, then clients could interactively choose to refine thequality associated with specific spatial regions over specific periodsof time, according to their level of interest. In a training video, forexample, a remote client could interactively “revisit” certain segmentsof the video and continue to stream higher quality information for thesesegments from the server, without incurring any delay.

To satisfy the needs of applications such as that mentioned above, lowbit-rate subsets of the video must be visually intelligible. Inpractice, this means that most of the available bits will be devoted toa low bit-rate portion of the video are likely to contribute to thereconstruction of the video at a reduced frame rate, since attempting torecover the full frame rate video over a low bit-rate channel willresult in unacceptable deterioration of the spatial details within eachframe. In order to achieve smooth quality scalability within acompressed video sequence which also offers frame rate scalability, thedetails required to recover higher frame rates must contribute to therefinement of a model which involves motion sensitive temporalinterpolation.

Without temporal interpolation, missing frames cannot be introduced intoa low rate video sequence without first augmenting their spatialfidelity to a level commensurate with the frames already available, andthis implies a large discontinuous jump in the amount of informationwhich must be provided to the decoder in order to smoothly increase thereconstructed video quality. Continuing this line of argument, we seethat motion information is important to highly scalable videocompression; moreover, the motion itself, must be represented in amanner which can be scaled, according to the temporal resolution (framerate), spatial resolution and quality of the sample data.

Motion Adaptive Transforms Based on Wavelet Lifting

The present invention is best appreciated in the context of an earlierinvention, which is the subject of WO02/50772. This earlier patentapplication describes a method for modifying the individual liftingsteps in a lifting implementation of a temporal wavelet decomposition,so as to compensate for the effects of motion. This method has thefollowing advantageous properties: 1) the motion sensitive transform maybe perfectly inverted, in the absence of any compression artefacts; 2)the low temporal resolution subsets of the wavelet hierarchy offer highspatial fidelity so that the transform allows excellent frame ratescalability; 3) the high pass temporal detail subbands produced by thetransform have very low energy, allowing high compression efficiency; 4)in the absence of motion, the transform reduces to a regular waveletdecomposition along the temporal axis; and 5) in the presence of locallytranslational motion, the transform is equivalent to applying a regularwavelet decomposition along the motion trajectories.

To assist in the present discussion, we briefly summarise the key ideasbehind this earlier invention. Any two-channel FIR subband transform canbe described as a finite sequence of lifting steps [W. Sweldens, “Thelifting scheme: A custom-design construction of biorthogonal wavelets,”Applied and Computational Harmonic Analysis, vol 3, pp 196-2000, April1996]. It is instructive to begin with an example based upon the Haarwavelet transform. Up to a scale factor, this transform may be realisedin the temporal domain, through a sequence of two lifting steps, as

h_(k)[n] = x_(2k + 1) − x_(2k)[n]${1_{k}\lbrack n\rbrack} = {x_{2k} + {\frac{1}{2}{h_{2k}\lbrack n\rbrack}}}$

where x_(k)[n]∝x_(k)[n₁,n₂] denotes the samples of frame k from theoriginal video sequence and h_(k)[n]∝h_(k)[n₁,n₂] andl_(k)[n]∝l_(k)[n₁,n₂] denote the high-pass and low-pass subband frames.

l_(k)[n] and h_(k)[n] correspond to the scaled sum and the difference ofeach original pair of flames. An example is shown in FIG. 1A.Since-motion is ignored, ghosting artefacts are clearly visible in thelow-pass temporal subband, and the high-pass subband, frame hassubstantial energy.

Now let W_(k1→k2) denote a motion-compensated mapping of frame k1 ontothe coordinate system of frame k2, so thatW_(k1→k2)(x_(k1)[n]≈x_(k2)[n]) for all n. The lifting steps are modifiedas follows.

$\begin{matrix}{{h_{k}\lbrack n\rbrack} = x_{{2k} + {1{\lbrack n\rbrack}} - W_{{2k}\rightarrow{{2k} + {1{{(x_{2k})}{\lbrack n\rbrack}}}}}}} & (1) \\{{l_{k}\lbrack n\rbrack} = {{x_{2k}\lbrack n\rbrack} + {\frac{1}{2}W_{{{2k} + 1}\rightarrow{2{k{({h_{k}{\lbrack n\rbrack}})}}}}}}} & (2)\end{matrix}$

Note that W_(2k→2k+1) and W_(2k+1→2k) represent forward and backwardmotion mappings, respectively. The high-pass subband frames correspondto motion-compensated residuals. These will be close to zero in regionswhere the motion is accurately modelled. The result is shown in FIG. 1B.

The framework described above is readily extended to any two-channel FIRsubband transform, by motion-compensating the relevant lifting steps.

We demonstrate this in the important case of the biorthogonal 5/3wavelet transform [D. Le Gall and A. Tabatabai, “Sub-band coding ofdigital images using symmetric short kernal filters and arithmeticcoding techniques,” IEEE International Conference on Acoustics, Speechand Signal Processing, vol. 2, pp 761-764, April 1988]. As before,x_(2k)[n] and x_(2k+1)[n] denote the even and odd indexed frames fromthe original sequence. Without motion, the 5/3 transform may beimplemented by alternatively updating each of these two framesubsequences, based on filtered versions of the other sub-sequence. Thelifting steps are

${h_{k}\lbrack n\rbrack} = {{x_{{2k} + 1}\lbrack n\rbrack} - {\frac{1}{2}\left( {{x_{2k}\lbrack n\rbrack} - {x_{{2k} + 2}\lbrack n\rbrack}} \right)}}$${l_{k}\lbrack n\rbrack} = {{x_{2k}\lbrack n\rbrack} + {\frac{1}{4}\left( {h_{k - 1}\lbrack n\rbrack} \right)}}$

As before, we introduce motion warping operators within each liftingstep, which yields the following

$\begin{matrix}{{h_{k}\lbrack n\rbrack} = {{x_{{2k} + 1}\lbrack n\rbrack} - {\frac{1}{2}\left( {{{W_{{2k}\rightarrow{{2k} + 1}}\left( x_{2k} \right)}\lbrack n\rbrack} + {{W_{{{2k} + 2}\rightarrow{{2k} + 1}}\left( x_{{2k} + 2} \right)}\lbrack n\rbrack}} \right)}}} & (3) \\{{l_{k}\lbrack n\rbrack} = {{x_{2k}\lbrack n\rbrack} + {\frac{1}{4}\left( {{{W_{{{2k} - 1}\rightarrow{2k}}\left( h_{k - 1} \right)}\lbrack n\rbrack} + {{W_{{{2k} + 1}\rightarrow{{2k} + 1}}\left( h_{k} \right)}\lbrack n\rbrack}} \right)}}} & (4)\end{matrix}$

FIG. 2 demonstrates the effect of these modified lifting steps. Thehighpass frames are now essentially the residual from a bidirectionalmotion compensated prediction of the odd-indexed original frames. Whenthe motion is adequately captured, these high-pass frames have littleenergy and the low-pass frames have excellent spatial fidelity.

Counting the Cost of Motion

In the example of the Haar transform, given above, two separate motionmapping operators, W_(2k→2k+1) and W_(2k+1→2k), are required to processevery pair of frames, x_(2k)[n] and x_(2k+1)[n]. Their respective motionparameters must be transmitted to the decoder. To provide a largernumber of temporal resolution levels, the transform is re-applied to thelow-pass subband frames, lk[n], for which motion mapping operatorsW_(4k→4k+2) and W_(4k+2→4k) are required for every four frames.Continuing in this way, an arbitrarily large number of temporalresolutions may be obtained, using

$\frac{2}{2} + \frac{2}{4} + \frac{2}{8} + {\ldots \mspace{14mu} {.2}}$

motion fields per original frame.

For the example of the 5/3 transform, also given above, four motionmapping operators, W_(2k→2k+1), W_(2k→2k−1), W_(2k+1→2k) and W_(2k−1→2k)are required for every pair of frames (indexed by k), for just one levelof temporal decomposition. Continuing the transformation to anarbitrarily large number of temporal resolutions involves approximately4 motion fields per original video frame.

The cost of estimating, coding and transmitting the above motion fieldscan be substantial. Moreover, this cost may adversely affect thescalability of the entire compression scheme, since it is notimmediately clear how to progressively refine the motion fields withoutdestroying the subjective properties of the reconstructed video when themotion is represented with reduced accuracy.

The previous invention clearly reveals the fact that any number ofmotion modelling techniques are compatible with the motion adaptivelifting transform, and also recommends the use of continuouslydeformable motion models such as those associated with, triangular orquadrilateral meshes (see, for example, Y. Nakaya and H. Harashima,“Motion compensation based on spatial transformations,” IEE Trans. Circ.Syst. For Video Tech., Vol. 4, pp 339-367, June 1994). However, noparticular solution is presented to the difficulties described above.

SUMMARY

Accordingly, in one aspect, the present invention provides a method forincrementally coding and signalling motion information for a videocompression system involving a motion adaptive transform and embeddedcoding of transformed video samples, said method comprising the stepsof: (a) producing an embedded bit-stream, representing each motion fieldin coarse to fine fashion; and (b) interleaving incrementalcontributions from said embedded motion fields with incrementalcontributions from said transformed video samples.

The present invention also provides a system for incrementally codingand signalling motion information for a video compression systeminvolving a motion adaptive transform and embedded coding of transformedvideo samples, said system comprising: (a) means for producing anembedded bit-stream, representing each motion field in coarse to finefashion; and (b) means for interleaving incremental contributions fromsaid embedded motion fields with incremental contributions from saidtransformed video samples.

Thus, because each motion field is represented in coarse to fine fashionand interleaved with the video data bit-stream, the accuracy requiredfor motion representation can be balanced with the accuracy of thetransformed sample values which may be recovered from the bit-stream.Therefore, a fully scalable video bit-stream may be progressivelyrefined, both in regard to its quantised sample representations and inregard to its motion representation.

Preferably, the embedded motion field bit-stream is obtained by applyingembedded quantization and coding techniques to the motion fieldparameter values.

Preferably, the embedded motion field bit-stream is obtained by codingthe node displacement parameters associated with a triangular meshmotion model on a coarse to fine grid, each successive segment of theembedded bit-stream providing displacement parameters for node positionswhich lie on a finer grid than the previous stage, all coarser grids ofnode positions being subsets of all finer grids of node points.

Preferably, a coarse to fine motion representation is obtained by firsttransforming the motion parameters and then coding the transformcoefficients using embedded quantization and coding techniques.

Preferably, the motion parameters are transformed by applying spatialdiscrete wavelet transforms and/or temporal transforms thereto.

Preferably, the spatial and/or temporal transforms are reversibleinteger-to-integer transforms, suitable for lossless compression.

Preferably, the embedded motion bit-streams are arranged into a sequenceof quality layers, and the transformed video samples are also encodedinto embedded bit-streams which are arranged into a separate sequence ofquality layers.

Preferably, said interleaving of the contributions from the embeddedmotion bit-streams and from the transformed video samples is performedin a manner which minimizes the expected distortion in the reconstructedvideo sequence at each of a plurality of compressed video bit-rates.

Preferably, the measure of distortion is Mean Squared Error. Preferably,the measure of distortion is a weighted sum of the Mean Squared Errorcontributions from different spatial frequency bands, weighted accordingto perceptual relevance factors.

Preferably, the distortion associated with inaccurate representation ofthe motion parameters is determined using an estimate of the spatialpower spectrum of the video source.

Preferably, the distortion associated with inaccurate representation ofthe motion parameters is determined using information about the spatialresolution at which the video bit-stream is to be decompressed.

Preferably, the power spectrum of the video source is estimated usingspatio-temporal video sample subbands created during compression.

Preferably, the proportions of contributions from said embedded motionfields and said transformed video samples in the embedded bit-stream isdetermined on the basis of a plurality of tables associated with eachframe, each table being associated with a spatial resolution at whichthe video bit-stream is to be decompressed. In the embodiment whereinthe embedded motion bit-streams and the transformed video, samples areeach encoded as a series of quality layers, the tables identify thenumber of motion quality layers which are to be included with eachnumber of video sample quality layers.

The preferred structure of the motion representation allowsrate-distortion optimal algorithms to balance the contributions ofmotion information and sample accuracy, as it is being included into anincrementally improving (or layered) compressed representation. Whilerate-distortion optimisation strategies for balancing motion and sampleaccuracy have been described in the literature, those algorithms wereapplicable only to static optimisation of a compressed bit-stream for asingle target bit-rate. The preferred embodiment of the presentinvention allows for the rate-distortion optimised balancing of motionand sample accuracy to be extended to scalable content in which thetarget bit-rate cannot be known a priori.

According to a further aspect of the present invention, a method forestimating and signalling motion information for a motion adaptivetransform based on temporal lifting steps, comprises the steps of (a)estimating and signalling motion parameters describing a first mappingfrom a source frame onto a target frame within one of the lifting steps;and (b) inferring a second mapping between either said source frame orsaid target frame, and another frame, based on the estimated andsignalled motion parameters associated with said first mapping.

The present invention also provides a system for estimating andsignalling motion information for a motion adaptive transform based ontemporal lifting steps, said system comprising: (a) means for estimatingand, signalling motion, parameters describing a first mapping from asource frame onto a target frame within one of the lifting steps; and(b) means for inferring a second mapping between either said sourceframe or said target frame, and another frame, based on the estimatedand signalled motion parameters associated with said first mapping.Accordingly, the number of motion fields which must be signalled to thedecompressor can be reduced, as some motion fields can be inferred fromothers.

For instance, in one embodiment said second mapping is the reciprocalmapping from said target frame to said source frame, for use withinanother one, of the lifting steps. Preferably, said reciprocal mappingis the inverse of the first mapping.

Thus, the preferred embodiment provides a method for estimating andrepresenting only one of the motion fields in each pair, W_(2k→2k+1) andW_(2k+1→2k), or W_(2k→2k−1), and W_(2k−1→2k). Such pairs of motionfields will be known here as “reciprocal pairs.” This allows the totalamount of motion information to be reduced to one motion field per framefor the Haar case, and 2 motion fields per frame for the 5/3 case. It isfound that collapsing reciprocal pairs to a single motion field, fromwhich the pair is recovered, actually improves the properties of themotion adaptive transform, resulting in increased compressionefficiency, even when the benefits of reduced motion cost are not takeninto account.

In one embodiment, the motion parameters of said first mappingcorrespond to deformable triangular mesh motion model. Preferably, saidreciprocal mapping is inferred by inverting the affine transformationsassociated with the triangular mesh used to represent said firstmapping.

In another embodiment, the motion parameters of said first mappingcorrespond to a block displacement motion model.

Preferably, said motion adaptive transform involves multiple stages oftemporal decomposition, corresponding to different temporal frame rates.

Preferably, motion parameters at each temporal resolution are deducedfrom original video frames.

In one embodiment said second mapping is a mapping between frames at alower temporal resolution than said first mapping, and said secondmapping is inferred by compositing the first mapping with at least onefurther mapping between frames at the higher temporal resolution.

This embodiment enables all of the required motion fields at lowertemporal resolutions (higher temporal displacements) to be derived froman initial set of frame-to-frame motion fields. Thus, the compressorneed only estimate the motion between each successive pair of frames,x_(k)[n] and x_(k+1)[n]. This substantially reduces the cost in memoryand computation of the motion estimation task, without significantlyaltering the compression efficiency or other properties of the motionadaptive transform.

In another embodiment, said second mapping is a mapping between framesat a higher temporal resolution than said first mapping, and said secondmapping is inferred by compositing the first mapping with at least onefurther mapping at the higher temporal resolution. For example,preferably the higher resolution is double said lower resolution, andalternate mappings at the higher temporal resolution are explicitlysignalled to a decompressor, the remaining mappings at the highertemporal resolution being replaced by the mappings inferred bycompositing the lower resolution mappings with respective higherresolution mappings. Preferably said replaced mappings are used withinthe lifting steps of said motion adaptive transform, in place of theoriginally estimated mappings which were replaced.

This further reduces the motion information to 1 motion field per videoframe, even for the 5/3 transform. The method of this embodiment has theproperty that the motion representation is temporally scalable. Inparticular, only one motion field must barnacle available to the decoderfor each video frame which it can reconstruct, at any selected temporalresolution. This method involves judicious compositing of the forwardand backward motion fields from different temporal resolution levels andis compatible with the efficient motion estimation method describedabove, of compositing motion fields at higher resolutions to obtainmotion fields at lower resolutions.

Preferably said replaced mappings are refined with additional motionparameters, said refinement parameters being signalled for use indecompression, and said replaced and refined mappings being used withinthe lifting steps of said motion adaptive transform, in place of theoriginally estimated mappings which were replaced.

Preferably, inversion or composition of motion transformations isaccomplished by applying said motion transformations to the nodepositions of a triangular mesh motion model, the composited or invertedmotion transformation being subsequently applied by performing theaffine transformations associated with said mesh motion model.

Preferably, the source frame is partitioned into a regular mesh and theinversion or composition operations are applied to each node daleregular mesh to find a corresponding location in the target frame, thecomposited or inverted motion transformation being subsequently appliedby performing the affine transformations associated with said meshmotion model. This is a particularly efficient computational method forperforming the various motion field transformations required by otheraspects of the invention. These methods are preferably replicated atboth the compressor and the decompressor, if the transform is to remainstrictly invertible.

BRIEF DESCRIPTION OF THE DRAWINGS

The various features, advantages and other uses of the present apparatuswill become more apparent by referring to the following detaileddescription and drawing in which:

FIG. 1A is Embodiments of the invention will now be described withreference to the accompanying drawings, in which:

FIG. 1A illustrates the lifting steps for the Haar temporal transform;

FIG. 1B illustrates a motion adaptive modification of the lifting stepsfor the Haar temporal transform;

FIG. 2 illustrates the lifting steps for a motion adaptive 5/3 temporaltransform;

FIG. 3 illustrates a triangular mesh motion model;

FIG. 4 illustrates schematically the compositing of two motion fields ata higher temporal resolution to create one at a lower resolution; and

FIG. 5 illustrates schematically the compositing of motion fields in oneembodiment of a temporally scalable motion representation for the motionadaptive 5/3 lifting transform.

DETAILED DESCRIPTION 1st Aspect: Reciprocal Motion Fields

While the invention has been described in connection with what ispresently considered to be the most practical and preferred embodiment,it is to be understood that the invention is not to be limited to thedisclosed embodiments but, on the contrary, is intended to cover variousmodifications and equivalent arrangements included within the spirit andscope of the appended claims, which scope into be accorded the broadestinterpretation, so as to encompass all such modifications and equivalentstructures as is permitted under the law.

A natural strategy for estimating the reciprocal motion fields,W_(2k→2k+1) and W_(2k+1→2k), would be to determine the parameters forW_(2k→2k+1) which minimise some measure (e.g., energy) of the mappingresidual x_(2k+1)−W_(2k→2k+1)(x_(2k)) and to separately determine theparameters for W_(2k+1→2k) which minimise some measure of its residualsignal, x_(2k)−W_(2k+1→2k)(x_(2k+1)). In general, such a procedure willlead to parameters for W_(2k→2k+1), which cannot be deduced from thosefor W_(2k+1→k2) and vice-versa, so that both sets of parameters must besent to the decoder.

It turns out that only one of the two motion fields must be directlyestimated. The other can then be deduced by “inverting” the motion fieldwhich was actually estimated. Both the compressor and the decompressormay perform this inversion so that only one motion field must actuallybe transmitted.

True scene motion fields cannot generally be inverted, due to thepresence of occlusions and uncovered background. One would expect,therefore, to degrade the properties of the motion adaptive transform(e.g., compression performance, or quality of the low temporalresolution frames) by replacing W_(2k→2k+1) with an approximate inverseof W_(2k+1→2k) or vice-versa.

It turns out, however, that the opposite is the case. Rather thandegrading the transform, representing each reciprocal pair with only onemotion field actually improves the compression efficiency and thequality of the low temporal resolution frames.

An explanation for the above phenomenon is given in. A. Seeker and D.Taubman, “Lifting-based invertible motion adaptive transform (LIMAT)framework for highly scalable video compression”, accepted to appear inIEEE Trans. Image Proc., 2003, a copy of which is available atwww.ee.unsw.edu.an/.about.taubman/. Briefly, the excellent properties ofthe motion adaptive temporal lifting transform are closely linked to thereciprocal relationship between the pairs, W_(2k→2k+1) and W_(2k+1→2k),and W_(2k→2k−1) and W_(2k−1→2k). If the frame warping operationsdescribed by each pair are truly inverses of one another, the motionadaptive transform is equivalent to a one-dimensional DWT, applied alongthe underlying motion trajectories. If they are not inverses of oneanother, this desirable characteristic is lost, no matter how well theyare able to minimise motion compensated residuals.

According to the first aspect of the present invention, only one motionfield from each reciprocal pair should be directly estimated andcommunicated to the decompressor. Unless otherwise prohibited (e.g., bythe later aspects of the invention), it is mildly preferable to directlyestimate and communicate the parameters of the motion field which isused in the first (predictive) lifting step. This is the lifting stepdescribed by equations (1) and (3), for the Haar and 5/3 cases,respectively.

Inversion of Triangular Mesh Motion Models

Where the motion is represented by a continuously deformable triangularmesh [Y. Nakaya and H. Harashima, “Motion compensation based on spatialtransformations”, IEEE Trans. Circ. Syst. For Video Tech., vol. 4, pp339-367, June 1994], the affine motion which describes the deformationof each triangle in. W_(2k→2k+1) or W_(2k→2k−1) may be directly invertedto recover W_(2k+1→2k) and W_(2k−1→2k), respectively. A triangular-meshmodel for motion field W_(k1+k2) involves a collection of nodepositions, {ti} in the target frame, x_(k2) together with the locations,{si} of those same node positions, as they appear in the source frame,x_(kl). Although scene adaptive meshes have been described, in thepreferred embodiment of the invention the target node positions, {ti},are fixed, and the motion field is parameterized by the set of nodedisplacements, {si-ti}. The target frame, x_(k2), is partitioned into acollection of disjoint triangles; whose vertices correspond to the nodepositions. Since the partition must cover the target frame, some of thetarget node positions must lie on the boundaries of the frame. Anexample involving a rectangular grid of target node vertices is shown inFIG. 3.

As suggested by the figure, it is convenient to write {Δ_(j)} for theset of target frame triangles. Let t_(j,o), t_(j,1) and t_(j,2) denotethe vertices of target triangle Δ_(j). The triangular mesh then maps thesource triangle, Δ′_(j), described by the vertices s_(j,o), s_(j,2) ands_(j,2) onto target triangle Δ′_(j), The motion map itself is describedby an affine transformation. Specifically, for each location, tεΔ_(j),within the target frame, the corresponding location, s, within thesource frame is given by the affine equation

s=A _(j) t+b _(j)

where t, s and b_(j) are regarded as column vectors, A_(j) is a 2×2matrix; A_(j) and b_(j) may be deduced from the motion parameters, usingthe fact that t_(j,i) must map to s_(j,i) for each i=0, 1, 2. Of course,s does not generally lie on an integer grid, and so the source framemust be interpolated, using any of a number of well-known methods, torecover the value of (W_(kl→k2)(x_(kl)))[t].

In the simplest case, whenever a target node position, t_(i), lies onthe boundary of frame) x_(k2), the corresponding source node position,s_(i), is constrained to lie on the same boundary of frame x_(kl), asdepicted in FIG. 3. In this case, the source triangles, Δ′_(j),completely cover the source frame and so each location, s, in framex_(kl), may be associated with one of the triangles, Δ^(i) _(j) andhence mapped back onto the target frame though the inverse affinerelation

t=A _(j) ⁻¹(s−b _(j))

In this way, the value of (W_(k2→k1)(x_(k2)))[s] may be found for eachlocation, s, by interpolating frame x_(k2) to the location, t.

Constraining boundary nodes, t_(i), map to nodes, on the same boundary,tends to produce unrealistic motion fields in the neighbOurhOod of theframe boundaries, adversely affecting the ability of the mesh to tracktrue scene motion trajectories. For this reason, the preferredembodiment of the invention does not involve any such constraints. Inthis case, the source triangles, Δ′_(j) will not generally cover framex_(k1), and inversion of the affine transformations yields values for(W_(k2→k1)(x_(k2)))(s) only when s lies within one of the sourcetriangles, Δ′_(j). For locations s which do not belong to any of thesource triangles, Δ′_(j), any of a number of policies may be described.As a simple example, the nearest source triangle, Δ′_(j), to s may befound and its affine parameters used to find a location t in framex_(k2).

An alternative approach is to first extrapolate the mesh to one which isdefined over a larger region than that required by the forward motionfield W_(k1→k2). So long as this region is large enough to cover thesource frame, each location s in frame x_(k1) will belong to some sourcetriangle within the extrapolated mesh and the corresponding affine mapcan be inverted to find the location t in frame x_(k2). In the preferredembodiment of this approach, the node vector t_(e)−s_(e) at eachextrapolated node position n_(e) in frame x_(k2), is obtained by linearextrapolation of two node vectors, t_(b)−s_(b) and t_(o)−s_(o), havingcorresponding node positions n_(b) and n_(o). Here, the extrapolatednode position n_(e) is outside the boundaries of frame x_(k2), n_(b) isthe location of the nearest boundary node to n_(e), andn_(o)=2n_(b)−n_(e) is the mirror image of n, through the boundary node,n_(b). The extrapolated node vectors are not explicitly communicated tothe decoder, since it extrapolates them from the available interior nodepositions, following the same procedure as the encoder.

“Inversion” of Block-Displacement Motion Models

Triangular mesh models are particularly suitable for the recovery of areverse motion field, W_(k2→k1), from its forward counterpart W_(k1→k2).Most significantly, the transformation between target locations, t, andsource locations, s, is continuous over the whole of the target frame.This is a consequence of the fact that the affine transformation mapsstraight lines to straight lines.

Block displacement models, however, are more popular for videocompression due to their relative computational simplicity. A blockdisplacement model consists of a partition of the target frame intoblocks, {B_(i)}, and a corresponding set of displacements, {δ_(i)},identifying the locations of each block within the source frame.

Unlike the triangular mesh, block displacement models represent themotion field in a discontinuous (piecewise constant) manner. As aresult, they may not properly be inverted. Nevertheless, when reciprocalpairs of motion maps, W_(k1→k2) and W_(k2→k1), use block displacementmodels, it is still preferable to estimate and transmit only one of thetwo motion fields to the decoder, inferring the other through anapproximate inverse relationship. Since displacements are usually small,it is often sufficient simply to reverse the sign of the displacementvectors, {δ_(i)}, when forming W_(k2→k1) from W_(k1→k2) or vice-versa.

2nd Aspect: Compositing of Simple Motion Fields.

For high energy compaction and low temporal resolution frames with highfidelity, it is essential to have accurate motion mappings for eachlevel of a multi-resolution temporal subband decomposition. Thetransform consists of a sequence of stages, each of which produces alow- and a high-pass temporal subband sequence, from its input sequence.Each stage in the temporal decomposition is applied to the low-passsubband sequence produced by the previous stage.

Since each stage of the temporal decomposition involves the same steps,one might consider applying an identical estimation strategy within eachstage, estimating the relevant motion fields from the frame sequencewhich appears at the input to that stage. The problem with such astrategy is that estimation of the true motion, based on subband frames,may be hampered by the existence of unwanted artefacts such as ghosting.Such artefacts can arise as a result of model failure or poor motionestimation, in previous stages of the decomposition.

To avoid this difficulty, it is preferred to perform motion estimationon the appropriate original frames instead of the input frames to thedecomposition stage in question. For example, in the second stage oftemporal decomposition it is more effective to estimate the motionmapping W^(k1→k2) ⁽¹⁾ between subband frames l_(k1) ₍₁₎ [n] and l_(k2)₍₂₎ [n], by using the corresponding original frames x_(k1)[n] andx_(k2)[n]. Similarly, in the third stage, it is more effective toestimate the motion mapping W_(k1→k2) ₍₂₎ between subband frames l_(k1)₍₂₎ _([n]) and l_(k2) ₍₂₎ _([n]), by using the corresponding originalframes x_(4k1)[n] and x_(4k2)[n]. To clarify the notation being usedhere, it is noted that the first stage of decomposition employs motionmappings W_(k1→k2) _((o)) , producing low and high-pass subband frames,l_(k) ₍₁₎ [n] and h_(k) ₍₁₎ [n]

After several levels of subband decomposition, the temporal displacementover which motion estimation must be performed will, span many originalframes. For example, in the fifth level of decomposition the actualtemporal displacement between neighbouring subband frames is 16 timesthe original frame displacement. At a typical frame rate of 30 framesper second (fps), this corresponds to more than half a second of video.

Motion estimation is generally very difficult over large temporaldisplacements due to the large possible-range of motion. This complexitycan be reduced by using knowledge of motion mappings already obtained inprevious levels of the decomposition. For example, as described byequations (3) and (4), the first stage of decomposition with the 5/3kernel involves estimation of W_(2k→2k+1) _((o)) and W_(2k+2→2k+1)_((o)) . These may be composited to form an initial approximation forW_(k→k+1) ₍₁₎ , which is required for the second stage of decomposition.This is shown in FIG. 4, where the arrows indicate the direction of themotion mapping. It is often computationally simpler to create compositemappings from source mappings that have the same temporal orientation,as suggested in the figure. If necessary, the source mappings can beinverted to achieve this. However, it is preferable to directly estimatesource mappings, having the same direction as the composite mapping.

The initial approximation, formed by motion, field composition in themanner described above, can be refined based on original video data,using motion estimation procedures well known to those skilled in theart. It turns out, however, that the method of compositing motion fieldswith a frame displacement of 1 to produce motion fields corresponding tolarger frame displacements often produces highly accurate motionmappings; that do not need any refinement. In some cases the compositmappings lead to superior motion adaptive transforms than motionmappings formed by direct estimation, or with the aid of refinementsteps. The motion field composition method described, here can berepeated throughout the temporal decomposition hierarchy so that all themappings for the entire transform can be derived from the frame to framemotion fields estimated in the first stage.

The composition method described above eliminates a significant portionof the computational load associated with direct estimation of therequired motion fields. A total of one motion mapping must be estimatedfor each original frame, having a temporal displacement of only oneframe. This is sufficient to determine the complete set of motionmappings for the entire transform.

This method is independent of the particular wavelet kernel on which thelifting framework is based; however, the effectiveness of thecomposition procedure does depend on the selected motion model. Anefficient method for performing the composition procedure is describedin the 4th aspect of this invention.

3rd Aspect: Efficient Temporally Scalable Motion Representation

An efficient temporally scalable motion representation should satisfytwo requirements. Firstly, at most one motion mapping per video frameshould be needed to reconstruct the video at any temporal resolution.This is consistent with the above observation that just one mapping perframe is sufficient to derive all mappings for the entire transform.

Secondly, the above property should apply at each temporal resolutionavailable from the transform. In particular, this means that the motioninformation must be temporally embedded, with each successively highertemporal resolution requiring one extra motion mapping per pair ofreconstructed video frames. This property allows the video content to bereconstructed at each available temporal resolution, without recourse toredundant motion information.

This aspect of the invention involves a temporally scalable motioninformation hierarchy, based on the method of motion field composition,as introduced in the description of the second aspect. Thisrepresentation achieves both of the objectives mentioned above.

The motion information hierarchy described here is particularlyimportant for motion adaptive lifting structures that are based onkernels longer than the simple Haar. Block transforms such as the Haarrequire only the motion information between every second pair ofconsecutive frames, at each stage of the decomposition. Therefore anefficient temporally scalable motion representation can be easilyachieved by transmitting a single motion mapping for every reciprocalpair.

It is generally preferable to use longer wavelet kernels such as the5/3. In fact, results given in A. Seeker and D. Taubman, “Lifting-basedinvertible motion adaptive transform (LIMAT) framework for highlyscalable video compression”, (accepted to appear in IEEE Trans. ImageProc., 2003) reveal that this can lead to considerable improvements inperformance.

The motion representation for two stages of the 5/3 transform is givenin FIG. 5. The mappings required to perform the lifting steps are againshown as arrows, where the i^(th) forward mapping in the j^(th)transform level is denoted F_(i) ^(j). The term “forward mapping” isapplied to those which approximate a current frame by warping a previousframe. Likewise, backward mappings, denoted B_(i) ^(j), correspond towarping a later frame to spatially align it with a current frame.Observe that the entire set of motion mappings depicted in FIG. 5 can berepresented using only F₁ ₂ ² and B₂ ₁ . Inverting F₁ ₂ produces thebackward mapping B₁ ₂ . The forward mapping F₁ ₁ is inferred bycompositing the upper-level forward mapping F¹ ² with the lower-levelbackward mapping B₂ ¹. The remaining mappings B₂ ¹ and F₂ ¹ arerecovered by inverting, F₁ ¹ and B² ² , respectively.

For scenes with rapid motion, composited fields such as F₁ ¹ in FIG. 5,may suffer from an accumulation of the model failure regions present inthe individual mappings. If so, the compressor may correct this bytransmitting an optional refinement fields, possibly based on directestimation using original data.

As mentioned, the case for the Haar wavelet is much simpler. Mappings F₂¹ Land B₂ ¹ are not required, so it is sufficient to code mappings F₁ ¹and F₁ ², recovering the corresponding backward motion fields byinversion. The methods described above can be applied recursively to anynumber of transform stages, and the total number of required mappings isupper bounded by one per original frame. Temporal scalability isachieved since reversing a subset of the temporal decomposition stagesrequires no motion information from higher resolution levels.

Evidently, a motion mapping between any pair of frames can be obtainedby a combination of composition and inversion operators involving thesequence of mappings F₁ ² and B_(2i) ¹. It follows that this motionrepresentation strategy is easily modified to encompass any waveletkernel.

4th Aspect: Efficient Implementation of Motion Field Transformations

A 4th aspect of the present invention describes an efficient method forperforming the motion field composition and inversion transformationsmentioned in previous aspects.

One possible way to represent a composited mapping is in terms of asequence of warpings through each individual mapping. Motioncompensation could be performed by warping the actual data through eachmapping in turn. However, this approach suffers from the accumulation ofspatial aliasing and other distortions that typically accompany eachwarping, step.

A second problem with this approach is that errors due to boundaryapproximations also accumulate over the sequence of mappings. Boundaryregions are prone to model failure, particularly when the sceneundergoes global motion such as camera panning.

To avoid these problems, each location in the target frame of thecomposite motion field may be mapped back through the various individualmappings to find its location in the source frame of the compositemotion field.

The preferred method, described here, however, is to construct atriangular mesh model for the composit motion field, deducing thedisplacements of the mesh node points by projecting them through thevarious component motion mappings. The triangular mesh model provides acontinuous interpolation of the projected node positions and can berepresented compactly in internal memory buffers. This method isparticularly advantageous when used in conjunction with triangular meshmodels for all of the individual motion mappings, since the framewarping machinery required to perform the motion adaptive temporaltransformation involves only one type of operation—the affinetransformation described previously.

Motion field inversion, may be performed using a similar strategy. Theinverted motion mapping is represented using a forward triangular meshmotion model, whose node displacements are first found by tracing themthrough the inverse motion field. The accuracy associated with bothcomposit and inverse motion-fields representations may be adjusted bymodifying the size of the triangular mesh grid. In the preferredembodiment of the invention, the mesh node spacing used for representingcomposit and inverse motion fields is no larger 8 frame pixels and nosmaller than 4 frame pixels.

5th Aspect: Successive Refinement of Motion and Sample Accuracy

In order to provide for scalable video bit-streams which span a widerange of bit-rates, from a few 10's of kilo-bits/s (kb/s) to 10's ofmega-bits/s (Mb/s), the accuracy with which motion information isrepresented must also be scaled. Otherwise, the cost of coding motioninformation would consume an undue proportion (all or more) of theoverall bit budget at low bit-rates and would be insufficient to providesignificant coding gain at high bit-rates. In the 3rd aspect above, amethod for providing temporally scalable motion information has beendescribed. In this 5th aspect, a method is described for further scalingthe cost of motion information, in a manner which is sensitive to boththe accuracy and the spatial resolution required of the reconstructedvideo sequence.

During compression, an accurate motion representation is determined andused to adapt the various lifting steps in the motion adaptivetransform. During decompression, however, it is not necessary to receiveexactly the same motion parameters which were used during compression.The motion parameters are encoded using an embedded quantisation andcoding strategy. Such strategies are now well known to those skilled inthe art, being employed in scalable image and video codecs such as thosedescribed in J. Shapiro, “Embedded image coding using zerotrees ofwavelet coefficients”, IEEE Trans. Sig. Proc., vol 41, pp 3445-3462,December 1993., D. Taubman and A. Zakhor, “Multi-rate 3-d subband codingof video”, IEEE Trans. Image Proc., vol. 3, pp. 572-588, September 1994,A. Said and W. Pearlman, “A new, fast and efficient image codee based onset partitioning in hierarchical trees”, IEEE Trans. Circ. Syst. ForVideo tech., pp. 243-250, June 1996, D. Taubman, “High performancescalable image compression with EBCOT”, IEEE Trans. Image Proc., vol. 9,pp. 1158-1170, July 2000. They allow the coded bit-stream to provide asuccessively more accurate representation of the information beingcoded. For the present purposes, this information consists of the motionparameters themselves, and each motion field, W_(k1→k2), is providedwith its own embedded bit-stream.

As an example of the way in which such an embedded motion representationmay be used, consider an interactive client-server application, in whichthe client requests information for the video at some particular spatialresolution and temporal resolution (frame rate). Based on thisinformation, the server determines the distortion which will beintroduced by approximating the relevant motion information with onlyL_(q) ^((M)) bits from the respective embedded bit-streams, where theavailable values for L_(q) ^((M)) are determined by the particularembedded quantisation and coding strategy which has been used. Let D_(q)^((M)) denote this distortion, measured in, terms of Mean Squared Error(MSE), or a visually weighted MSE. The values D_(q) ^((M)) may beestimated from the spatial-frequency power spectrum of the relevant,frames. Most notably, D_(q) ^((M)) depends not only on the accuracy withwhich the motion parameters are represented by the L_(q) ^((M)) bits ofembedded motion information, but also on the spatial resolution ofinterest. At lower spatial resolutions, less accuracy is required forthe motion information, since the magnitude of the phase shiftsassociated with motion error are directly proportional to spatialfrequency.

Continuing the example, above, the server would also estimate or knowthe distortion, D_(p) ^((s)), associated with the first L_(p) ^((s))bits of the embedded representation generated during scalable coding ofthe sample values produced by the motion adaptive transform. As alreadynoted, scalable sample data compression schemes are well known to thoseskilled in the art Assuming an additive model for these two differentdistortion contributions, the server balances the amount of informationdelivered for the motion and sample data components, following the usualLagrangian policy. Specifically, given a total budget of L^(max) bitsfor both components, deduced from estimates of the network transportrate, or by any other means the server finds the largest values of p_(λ)and q_(λ) such that

$\begin{matrix}{{\frac{{- \Delta}\; D_{p\; \lambda}^{(S)}}{\Delta \; L_{p\; \lambda}^{(S)}} \geq \lambda}{and}{\frac{{- \Delta}\; D_{q\; \lambda}^{(M)}}{\Delta \; L_{q\; \lambda}^{(M)}} \geq {\lambda.}}} & (5)\end{matrix}$

adjusting λ>0 so that L_(pλ) ^((s))+L_(qλ) ^((M)) is as large aspossible, while not exceeding L^(max). Here, ΔD_(p) ^((s))/ΔL_(p) ^((s))and ΔD^(q) ^((M)) /ΔL_(q) ^((M)) are discrete approximations to thedistortion-length slope at the embedded truncation points p (for sampledata) and q (for motion data) respectively.

The client-server application described above is only an example.Similar techniques may be used to construct scalable compressed videofiles which contain an embedded hierarchy of progressively higherquality video, each level in the hierarchy having its own balancebetween the amount of information contributed from the embedded motionrepresentation and the embedded sample data representation.

The strategy described above, whereby an embedded motion representationis produced by embedded quantisation and coding of the individual motionparameters, may be extended to include progressive refinement accordingto the density of the motion parameters themselves. To see how thisworks, suppose that every second row and every second column weredropped from the rectangular grid of node positions in the triangularmesh of FIG. 3. In this coarse mesh, motion parameters would be sentonly for the remaining node positions and the coarse triangular meshmodel induced by this information would represent a coarse approximationto the original motion model. Such approximations are readily includedwithin an embedded motion representation, from which an appropriatedistribution between the motion and sample data coding costs may againbe formed.

While rate-distortion optimisation strategies have previously beendescribed in the literature for balancing the costs of motion and sampledata information, this has not previously been done in a scalablesetting, where both the motion and the sample data accuracy areprogressively refined together.

While rate-distortion optimization strategies have previously been described in the literature for balancing the costs of motion and sampledata information, this has not previously been done in a scalablesetting, where both the motion and the sample data accuracy areprogressively refined together. There are, in our opinion, two principlereasons why progressively refined motion fields have not beeninvestigated in the past for video compression. Firstly, most existingvideo compression systems (e.g., those described by internationalstandards) employ motion compensated predictive coding, so if thedecoder were to use different motion parameters to the encoder, theirstates would progressively drift apart. This problem does not exist inthe context of motion adaptive wavelet transforms and, in particular,those based on the motion compensated lifting paradigm taught inW002/50772.

The second reason, why we believe others have not investigatedprogressively refillable motion for scalable video coding is that themotion information interacts in a complex manner with the video sampledata, making it more difficult to deduce the impact of motionquantization on system performance. The invention disclosed here,however, is inspired by the following interesting observation. Althoughthe interaction between motion errors and video sample data errors isgenerally complex, at all experimentally optimal combinations of themotion and sample data accuracy, this relationship simplifies and may beapproximately modeled using linear methods. In the ensuing sub-sections,we teach some specific methods for scalable motion coding and foroptimally balancing the distribution of motion information with videosample information.

Scalable Motion Coding Methods

As mentioned above, a variety of methods for embedded coding of data arewell known to those skilled in the art. Amongst these various methods,the authors' experimental investigations have suggested particularpreferred embodiments. Rather than coding the motion parametersdirectly, it is preferable to first subject the motion parameter fieldsto a spatial discrete wavelet transform (DWT). That is, the horizontalcomponents of each motion vector are treated as a two dimensional imageand the vertical components are similarly treated as a two dimensionalimage; each image is subjected to a spatial DWT and the transformcoefficients are then coded in place of the original motion vectors.

The use of a spatial wavelet transform is found to offer two chiefbenefits over coding the motion parameters directly. Firstly, thetransform typically produces a large number, of near-zero valuedcoefficients which can be quantized to zero with negligible error andthen efficiently encoded. Secondly, the DWT shapes the quantizationerrors incurred when the motion representation is scaled, and thisshaping is found to significantly reduce the reconstructed videodistortion incurred at any given level of motion quantization error. Inthe preferred embodiment, a reversible (integer-to-integer) spatial DWTis used to allow exact recovery of the originally estimated motionparameters from the encoded transform coefficients, which is useful athigh video bit-rates. Reversible wavelet transforms are well-known tothose skilled in the art. One example is the reversible 5/3 spatial DWTwhich forms part of the JPEG2000 image compression standard, IS 15444-1.

Temporal transformation of the motion parameter information can havesimilar benefits to spatial transformation, and the effects are found tobe somewhat complementary. That is, both the use of both a spatial DWTand a temporal transform together is recommended. In one particularembodiment, each pair of temporally adjacent motion fields is replacedby the sum and the difference of the corresponding motion vectors. Thesesums and differences may be interpreted as temporal low- and high-passsubbands.

Again, it is preferable to do this in a reversible manner which iscompatible with efficient lossless coding, since at high video bit-ratesit is best to preserve all of the estimated motion information. For thisreason, the operations of sum and difference mentioned above should bereplaced by the S-transform [V. I-leer and H. E. Reinfelder, “Acomparison of reversible methods for data compression”, Proc. SPIEconference, ‘Medical Imaging IV’, vol 1233, pp. 354-365, 1990].

As for the coding of motion transform coefficients, the preferredembodiments are those which use techniques derived from the generalclass of bit-plane coders. In particular, the highly efficient andfinely embedded fractional bit-plane coding techniques which form partof the JPEG2000 image compression standard are to be recommended. Ingeneral, each subband produced by the motion parameter transform ispartitioned into code-blocks, and each code-block is encoded using afractional bit-plane coder, producing a separate finely embeddedbit-stream for each motion subband code-block.

In many cases, there are insufficient motion parameters to justifydividing motion subbands into multiple code-blocks, but the code-blockpartitioning principles enshrined in the JPEG2000 standard can be usefulwhen compressing very large video frames, each of which has a largenumber of motion vectors. It general, then, the motion information isrepresented by a collection of code-blocks; each of which has a finelyembedded bit-stream which may be truncated to any of a variety of codedlengths.

A Layered Framework for Joint Scaling, of Motion and Video Sample Data

The EBCOT algorithm [D. Taubman, “High performance scalable imagecompression with EBCOT”, IEEE Trans. Image Proc., vol. 9, pp. 1158-1170,July 2000] represents an excellent framework for converting a largenumber of embedded code-block bit-streams, each with its own set oftruncation points, into a global collection of abstract “quality”layers. Each quality layer contains incremental contributions from eachcode-blocks embedded bit-stream, where these contributions are balancedin a manner which minimises the distortion associated with the overallrepresentation at the total bit-rates associated with the quality layer.By arranging the quality layers in sequence, one obtains a succession oftruncation points, at each of which the representation is as accurate asit can be, relative to the size of the included quality layers.

Although the interaction between motion errors and video sample errorsis non-trivial, it turns out that for combinations of motion and videosample bit-rates which are optimal, the relationship between motionerror and reconstructed video quality is approximately linear. We mayrepresent this linear relationship as

D _(x,M)≈Ψ_(R,S) D _(M)

where D_(M) denotes mean squared error in the motion vectors due totruncation of the embedded motion parameter code-block bit-streams, andD_(x,M) represents the total induced squared error in the reconstructedvideo sequence. The scaling factor, Ψ_(R,S), depends upon the spatialresolution at which the video signal is to be reconstructed and alsoupon the accuracy with which the video samples are represented. Inpreferred embodiments of the present invention, motion parameter qualitylayers are constructed from the embedded motion block bit-streams,following the EBCOT paradigm.

In view of the above relationship, and noting that the scaling factor,Ψ_(R,S), is substantially similar for all motion coefficient subbandsand code-blocks, the rate-distortion optimality of the layered motionrepresentation holds over a wide range of spatial resolutions and levelsof video sample quantization error. This is extremely convenient, sinceit means that the rate-distortion optimization problem expressed inequation (5) can be solved once, while constructing the motion qualitylayers, after which a video server or transcoder need only decide howmany motion layers are to be included in the video bit-stream for agiven spatial resolution and a given level of error in the video sampledata.

In preferred embodiments of the invention, the same layering strategy ofEBCOT is used to construct a separate set of rate-distortion optimalquality layers for the video sample data. These are obtained bysubjecting the temporal subbands produced by the motion-compensatedtemporal lifting steps to spatial wavelet transform, partitioning thespatio-temporal video subbands into their own code-blocks, andgenerating, embedded bit-streams for each video sample code-block. Thevideo sample quality layers then consist of incremental contributionsfrom the various video sample code-blocks, such at the video sampledistortion is as small as it can be, relative to the total size of thosequality layers. It turns out, most conveniently, that the generation ofrate-distortion optimal video sample quality layers is substantiallyindependent of the spatial resolution (number of resolution levels fromthe spatial video sample DWT which will be sent to the decoder) and thetemporal resolution (number of temporal subbands produced by the motioncompensated lifting steps which will be sent to the decoder). It alsoturns out that the optimality of the layer boundaries is approximatelyindependent of the level of motion distortion, at least for combinationsof motion and video sample bit-rates which are approximately optimal.

In summary, preferred embodiments of the invention produce a single setof motion quality layers and a single set of video sample qualitylayers. The layers are internally rate-distortion optimal over thetemporal interval within which they are formed. Since video streams canhave unbounded duration, we divide the time scale into epochs known as“frame slots” In each frame slot, a separate set of motion qualitylayers and video sample quality layers is formed. The optimizationproblem associated with equation (5) then reduces to that of balancingthe number of motion quality layers with the number of video samplequality layers which are sent to a decoder within each frame slot. Thesolution to this problem is dealt with below, but we note that itdepends on the parameter Ψ_(R,S) which is a function of both thespatial, resolution of interest to the decoder and the accuracy of thevideo sample data. Equivalently, for any particular number of videosample layers, p, the number of motion layers, q, which balances therate-distortion slopes of the motion and video sample information is afunction of both p and the spatial resolution of interest.

Methods for Optimizing the Distribution of Motion and Video Sample Data

In view of the preceding discussion, a complete implementation of thepreferred embodiment of the invention must provide a means for decidinghow many motion quality layers, q, are to be included with a subset ofthe video bit-stream which includes p video sample quality layers, giventhe spatial resolution R, at which the video content is to be viewed.The preferred way to do this is to include a collection of tables witheach frame slot, there being one table per spatial resolution which maybe of interest, where each table provides an entry for each number ofvideo sample quality layers, p, identifying the corresponding bestnumber of motion layers, q_(p). Depending upon the application, theremay be no need to send the table itself to a decoder.

A video server or transcoder, needing to meet a compressed lengthconstraint L_(max) Within each frame slot, can use these tables todetermine p and q_(p) which are jointly optimal, such that the totallength of the respective quality layers is as small as possible, but nosmaller than L_(max). It is then preferable to discard data from thep^(th) video sample quality layer, until the length target L_(max) issatisfied. This approach is preferable to that of discarding motiondata, since there is generally more video sample data. One way to buildthe aforementioned tables is to simply decompress the video at eachspatial resolution, using each combination of motion and sample qualitylayers, q and p, so as to find the value of p_(q) which maximizes theratio of distortion to total bit-rate in each frame slot, for each p. Ofcourse, this can be computationally expensive. Nevertheless, thisbrute-force search strategy is computationally feasible.

A preferred means to build the aforementioned tables is to use the factthat these tables depend only on the linear scaling factors, Ψ_(R,S).These scaling factors depend, in turn, on the power spectra of the videoframes which are reconstructed at each level of video sample error,i.e., at each video sample quality layer p. In the preferred embodimentof the invention, these power spectra are estimated directly from thevideo sample subband data during the compression process. We find, inpractice, that such estimation strategies can produce results almost asgood as the brute force search method described above, at a fraction ofthe computational cost.

1. A method for incrementally coding and signaling motion informationfor a video compression system involving a motion adaptive transform andembedded coding of transformed video samples using a computer, saidmethod comprising the steps of: (a) storing computer-readableinstructions in the computer which, when executed, produce an embeddedmotion field bit-stream, representing each motion field in coarse tofine fashion; and (b) storing computer-readable instructions in thecomputer which, when executed, interleave successive contributions fromsaid embedded motion field bit-stream with successive contributions fromsaid embedded coding of the transformed video samples.
 2. A system forincrementally coding and signaling motion information for a videocompression system involving a motion adaptive transform and embeddedcoding of transformed video samples, said system comprising: (a) meansfor producing an embedded motion field bit-stream, representing eachmotion field in coarse to fine fashion; and (b) means for interleavingsuccessive contributions from said embedded motion fields fieldbit-stream with successive contributions from said embedded coding ofthe transformed video samples.
 3. A system for estimating andcommunicating motion information required by a multi-frame encoding anddecoding system which involves a motion adaptive transform based ontemporal lifting steps, said system comprising: (a) means for estimatingand communicating motion parameters describing a first mapping from asource frame onto a target frame within one of the lifting steps; (b)means for inferring a second mapping within the encoding system betweeneither said source frame or said target frame, and another frame, basedon the estimated and signaled motion parameters associated with saidfirst mapping; and (c) means for inferring the second mapping within thedecoding system between either said source frame or said target frameand another frame, based on the communicated motion parametersassociated with said first mapping.
 4. The system of claim 2, where theembedded motion field bit-stream is obtained by applying embeddedquantization and coding techniques to motion field parameter values. 5.The system of claim 2, where the embedded motion field bit-stream isobtained by coding node displacement parameters associated with atriangular mesh motion model on a coarse to fine grid, each successivesegment of the embedded bit-stream providing displacement parameters fornode positions which lie on a finer grid than the previous stage, allcoarser grids of node positions being subsets of all finer grids of nodepoints.
 6. The system of claim 5, where a coarse to fine motionrepresentation is obtained by first transforming the motion parametersand then coding the transform coefficients using embedded quantizationand coding techniques.
 7. The system of claim 6, where the motionparameters are transformed by applying at least one of either spatialdiscrete wavelet transforms or temporal transforms thereto.
 8. Thesystem of claim 7, wherein at least one of either the spatial ortemporal transforms are reversible integer-to-integer transforms,suitable for lossless compression.
 9. The system of claim 2, wherein theembedded motion bit-streams are arranged into a sequence of qualitylayers, and the transformed video samples are also encoded into embeddedbit-streams which are arranged into a separate sequence of qualitylayers.
 10. The system of claim 2, where said interleaving of thecontributions from the embedded motion bit-streams and from thetransformed video samples is performed in a manner which minimizesexpected distortion in reconstructed video sequence at each of aplurality of compressed video bit-rates.
 11. The system of claim 10,where the measure of distortion is Mean Squared Error.
 12. The system ofclaim 10, where the measure of distortion is a weighted sum of the MeanSquared Error contributions from different spatial frequency bands,weighted according to perceptual relevance factors.
 13. The system ofclaim 10, where the distortion associated with inaccurate representationof the motion parameters is determined using an estimate of a spatialpower spectrum of the video source.
 14. The system of claim 10, wherethe distortion associated with inaccurate representation of the motionparameters is determined using information about a spatial resolution atwhich the video bit-stream is to be decompressed.
 15. The system ofclaim 13, where the spatial power spectrum of the video source isestimated using spatio-temporal video sample subbands created duringcompression.
 16. The system of claim 2, wherein proportions ofcontributions from said embedded motion fields and said transformedvideo samples in the embedded bit-stream are determined on the basis ofa plurality of tables associated with each frame, each table beingassociated with a spatial resolution at which the video bit-stream is tobe decompressed.
 17. The system of claim 16, wherein the embedded motionbit-streams and the transformed video samples are each encoded as aseries of quality layers and the tables identify a number of motionquality layers are to be included with each number of video samplequality layers.
 18. The system of claim 3, where said second mapping isthe reciprocal mapping from said target frame to said source frame, foruse within another one of the lifting steps.
 19. The system of claim 18,where said reciprocal mapping is the inverse of the first mapping. 20.The system of claim 19, where the motion parameters of said firstmapping correspond to a deformable triangular mesh motion model.
 21. Thesystem of claim 20, where said reciprocal mapping is inferred byinverting the affine transformations associated with the triangular meshused to represent said first mapping.
 22. The system of claim 3, wherethe motion parameters of said first mapping correspond to a blockdisplacement motion model.
 23. The system of claim 3, where said motionadaptive transform involves multiple stages of temporal decomposition,corresponding to different temporal frame rates.
 24. The system of claim23 where motion parameters at each temporal resolution are deduced fromoriginal video frames.
 25. The system of claim 23, wherein said secondmapping is a mapping between frames at a lower temporal resolution thansaid first mapping, and said second mapping is inferred by compositingthe first mapping with at least one further mapping between frames atthe higher temporal resolution.
 26. The system of claim 23, wherein saidsecond mapping is a mapping between frames at a higher temporalresolution than said first mapping, and said second mapping is inferredby compositing the first mapping with at least one further mapping atthe higher temporal resolution.
 27. The system of claim 26, wherein saidhigher resolution is double said lower resolution, and alternatemappings at the higher temporal resolution are explicitly signaled to adecompressor, the remaining mappings at the higher temporal resolutionbeing replaced by the mappings inferred by compositing the lowerresolution mappings with respective higher resolution mappings.
 28. Thesystem of claim 27, where said replaced mappings are used within thelifting steps of said motion adaptive transform, in place of theoriginally estimated mappings which were replaced.
 29. The system ofclaim 27, where said replaced mappings are refined with additionalmotion parameters, said refinement parameters being signaled for use indecompression, and said replaced and refined mappings being used withinthe lifting steps of said motion adaptive transform, in place of theoriginally estimated mappings which were replaced.
 30. The system ofclaim 3, where inversion or composition of motion transformations isaccomplished by applying said motion transformations to the nodepositions of a triangular mesh motion model, the composited or invertedmotion transformation being subsequently applied by performing theaffine transformations associated with said mesh motion model.
 31. Thesystem of claim 30, where the source frame is partitioned into a regularmesh and the inversion or composition operations are applied to eachnode of the regular mesh to find a corresponding location in the targetframe, the composited or inverted motion transformation beingsubsequently applied by performing the affine transformations associatedwith said mesh motion model.
 32. A non-transitory computer-readablestorage medium with an executable program stored thereon, wherein theprogram instructs the computer to implement the method of claim
 1. 33. Anon-transitory computer-readable storage medium with an executableprogram stored thereon, wherein the program instructs the computer toimplement the method of claim
 2. 34. The method of claim 1, where theembedded motion field bit-stream is obtained by applying embeddedquantization and coding techniques to motion field parameter values. 35.The method of claim 1, where the embedded motion field bit-stream isobtained by coding node displacement parameters associated with atriangular mesh motion model on a coarse to fine grid, each successivesegment of the embedded bit-stream providing displacement parameters fornode positions which lie on a finer grid than the previous stage, allcoarser grids of node positions being subsets of all finer grids of nodepoints.
 36. The method of claim 35, where a coarse to fine motionrepresentation is obtained by first transforming the motion parametersand then coding the transform coefficients using embedded quantizationand coding techniques.
 37. The method of claim 36, where the motionparameters are transformed by applying at least one of either spatialdiscrete wavelet transforms or temporal transforms thereto.
 38. Themethod of claim 37, wherein at least one of either the spatial ortemporal transforms are reversible integer-to-integer transforms,suitable for lossless compression.
 39. The method of claim 1, whereinthe embedded motion bit-streams are arranged into a sequence of qualitylayers, and the transformed video samples are also encoded into embeddedbit-streams which are arranged into a separate sequence of qualitylayers.
 40. The method of claim 1, where said interleaving of thecontributions from the embedded motion bit-streams and from thetransformed video samples is performed in a manner which minimizesexpected distortion in reconstructed video sequence at each of aplurality of compressed video bit-rates.
 41. The method of claim 40,where the measure of distortion is Mean Squared Error.
 42. The method ofclaim 40, where the measure of distortion is a weighted sum of the MeanSquared Error contributions from different spatial frequency bands,weighted according to perceptual relevance factors.
 43. The method ofclaim 40, where the distortion associated with inaccurate representationof the motion parameters is determined using an estimate of a spatialpower spectrum of the video source.
 44. The method of claim 40, wherethe distortion associated with inaccurate representation of the motionparameters is determined using information about a spatial resolution atwhich the video bit-stream is to be decompressed.
 45. The method ofclaim 43, where the spatial power spectrum of the video source isestimated using spatio-temporal video sample subbands created duringcompression.
 46. The method of claim 1, wherein proportions ofcontributions from said embedded motion fields and said transformedvideo samples in the embedded bit-stream are determined on the basis ofa plurality of tables associated with each frame, each table beingassociated with a spatial resolution at which the video bit-stream is tobe decompressed.
 47. The method of claim 46, wherein the embedded motionbit-streams and the transformed video samples are each encoded as aseries of quality layers and the tables identify a number of motionquality layers are to be included with each number of video samplequality layers.
 48. A method for estimating and communicating motioninformation required by a multi-frame encoding and decoding system whichinvolves a motion adaptive transform based on temporal lifting stepsusing a computer, said method comprising the steps of: (a) storingcomputer-readable instructions in the computer which, when executed,estimate and communicate motion parameters describing a first mappingfrom a source frame onto a target frame within one of the lifting steps;(b) storing computer-readable instructions in the computer which, whenexecuted, infer a second mapping within the encoding system betweeneither said source frame or said target frame, and another frame, basedon the estimated and communicated motion parameters associated with saidfirst mapping; and (c) storing computer-readable instructions in thecomputer which, when executed, infer the second mapping within thedecoding system between either said source frame or said target frameand another frame, based on the communicated motion parametersassociated with said first mapping.
 49. The method of claim 48, wheresaid second mapping is the reciprocal mapping from said target frame tosaid source frame, for use within another one of the lifting steps. 50.The method of claim 49, where said reciprocal mapping is the inverse ofthe first mapping.
 51. The method of claim 48, where the motionparameters of said first mapping correspond to a deformable triangularmesh motion model.
 52. The method of claim 51, where said reciprocalmapping is inferred by inverting the affine transformations associatedwith the triangular mesh used to represent said first mapping.
 53. Themethod of claim 48, where the motion parameters of said first mappingcorrespond to a block displacement motion model.
 54. The method of claim48, where said motion adaptive transform involves multiple stages oftemporal decomposition, corresponding to different temporal frame rates.55. The method of claim 54 where motion parameters at each temporalresolution are deduced from original video frames.
 56. The method ofclaim 54, wherein said second mapping is a mapping between frames at alower temporal resolution than said first mapping, and said secondmapping is inferred by compositing the first mapping with at least onefurther mapping between frames at the higher temporal resolution. 57.The method of claim 54, wherein said second mapping is a mapping betweenframes at a higher temporal resolution than said first mapping, and saidsecond mapping is inferred by compositing the first mapping with atleast one further mapping at the higher temporal resolution.
 58. Themethod of claim 57, wherein said higher resolution is double said lowerresolution, and alternate mappings at the higher temporal resolution areexplicitly signaled to a decompressor, the remaining mappings at thehigher temporal resolution being replaced by the mappings inferred bycompositing the lower resolution mappings with respective higherresolution mappings.
 59. The method of claim 58, where said replacedmappings are used within the lifting steps of said motion adaptivetransform, in place of the originally estimated mappings which werereplaced.
 60. The method of claim 58, where said replaced mappings arerefined with additional motion parameters, said refinement parametersbeing signaled for use in decompression, and said replaced and refinedmappings being used within the lifting steps of said motion adaptivetransform, in place of the originally estimated mappings which werereplaced.
 61. The method of claim 60, where inversion or composition ofmotion transformations is accomplished by applying said motiontransformations to the node positions of a triangular mesh motion model,the composited or inverted motion transformation being subsequentlyapplied by performing the affine transformations associated with saidmesh motion model.
 62. The method of claim 61, where the source frame ispartitioned into a regular mesh and the inversion or compositionoperations are applied to each node of the regular mesh to find acorresponding location in the target frame, the composited or invertedmotion transformation being subsequently applied by performing theaffine transformations associated with said mesh motion model.